Search system and method having civility score

ABSTRACT

A scoring system and method identifies personal attacks in a piece of audio content and generates a civility score for the piece of audio content that can differentiate between personal attacks and vernacular/casual banter. The piece of audio content may be a podcast.

PRIORITY CLAIMS/RELATED APPLICATIONS

This application is a continuation in part of and claims priority under35 USC 120 to U.S. patent application Ser. No. 18/220,437 filed Jul. 11,2023 and titled “Search System and Method Having Quality Scoring”, theentirety of which is incorporated herein by reference.

FIELD

The disclosure relates to search system and method that generates acivility score for content and in particular to a search system andmethod for podcasts that generates a civility score for podcasts thatmay be displayed to a user.

BACKGROUND

A podcast is a program made available in digital format for downloadover the Internet. For example, a podcast may be digital audio filesthat can be consumed by a person that downloads the podcasts. Examplesof the most popular types of podcasts include interview podcasts,conversational (co-hosted format) podcasts, educational podcasts, solopodcasts, non-fiction storytelling+news podcasts, podcast theater andbite-sized content or limited run podcast series.

Companies may want to present advertisements along with the podcast. Inorder to determine the right podcast for an advertisement, a brandmarketer needs a highly actionable way to measure brand suitability andmaximize brand affinity in podcast advertising. It would be desirable toprovide a tool by which advertisers can objectively measure the volumeand intensity of personal attacks within the nation's top podcasts.Trying to identify personal attacks in podcasts presents severalchallenges. First, because the language used in most podcasts may beinformal, sarcastic or ironic, differentiating between personal attacksand vernacular/casual banter is non-trivial. Second, speak or talk in apodcast is subject to individual interpretations and cultural norms anda discussion that one person perceives as offensive may not be offensiveto another person. Third, the variability in language and tone fordifferent speakers further complicates the process of identifyinginsults and personal attacks, especially if the listener is not familiarwith the speakers' background or communication style. These challengesresult from technical problems in known system and techniques thatcannot and do not identify personal attacks in podcasts.

Thus, it is desirable to provide a civility scoring system that usestechnology and provides a technical solution to identify personalattacks in a podcast and generates a civility score for each podcast andit is to this end that the disclosure is directed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a system that incorporates a civility scoreengine that identifies personal attacks in an audio piece of content andgenerates a civility score for each piece of audio content that may bereturned to the user as part of the search results;

FIG. 2 illustrates more details of the civility scoring engine thatidentifies personal attacks in audio content, such as a podcast, andgenerates a civility score for each piece of audio content;

FIG. 3 illustrates a method for identifying personal attacks andgenerating a civility score for a piece of audio content;

FIG. 4 illustrates an example of an LLM prompt used in the civilityscore engine in FIG. 2 ; and

FIG. 5 illustrates an example of a user interface that displays variousaudio content, a civility score and a score change.

DETAILED DESCRIPTION OF ONE OR MORE EMBODIMENTS

The disclosure is particularly applicable to a consumer facing searchengine that displays a civility score for each audio content, such aspodcasts, wherein a civility score generating engine (that may be partof the above search engine or may be a separate system that generatesthe civility scores) may generate the civility scores for each piece ofaudio content using ensemble artificial intelligence (AI)/machinelearning techniques so that the user can see whether a particular pieceof audio content has a particular civility score based on personalattacks found by the civility score engine in the particular piece ofaudio content. Alternatively, the civility score for podcasts or otheraudio content may be used by an advertisement system that placesadvertisements or recommends ad placement in audio content that allowsthe customers who wish to place ads for audio content to make the adplacement decision based in part of the certain civility scoreassociated with the piece of audio content. It should be understood thatthe disclosed system may be used for various different pieces of audiocontent (video with audio, etc.) and is not limited to podcasts. It alsowill be appreciated that the system and method may be implemented usingdifferent computer architectures such as a software as a service (SAAS)architecture or other known or yet to be developed computer systemarchitectures. Furthermore, the system may implement an applicationprogramming interface (API) so that any business/system may request andreceive the civility score generated for a particular piece of audiocontent. Furthermore, the system may be a standalone system accessedover the web by users as shown in FIG. 1 , but may also be embedded/partof a larger system. The system and method may be used to score any typeof audio content and return results although, for illustration purposes,podcasts will be discussed to illustrate the civility assessment. In afurther alternative embodiment, the generated civility score for a pieceof audio content may be used in a system in combination with the scoringdisclosed in U.S. patent application Ser. No. 18/220,437 filed Jul. 11,2023 (that is incorporated herein by reference) so that the scoringsystem may display pieces of content results to the user with thedocument quality scores and political leaning for written content (andpossible the audio content such as podcasts) and the civility scores forthe audio content.

The system and method disclosed below may have various technicalfeatures including proprietary processes for annotating, diversifyingand de-noising large training sets that is crucial for accounting forthe variability in cultural interpretations and norms, one or more largelanguage model (LLM) transformer architectures that focus not only onthe parts of the text that may contain attacks but also, the broadercontext, proprietary processes for assigning a civility score to anentire episode and in turn, show and proprietary prompt engineeringtechniques that corroborate domain-expert knowledge and enable the AItechnology to align with the target task's objectives to identifypersonal attacks in audio pieces of content and generate a civilityscore. These technical features provide a technical solution to atechnical problem of identifying personal attacks in audio content thatcannot be achieved by a human being. In summary, the disclosed systemand method provides many technical solutions and benefits over knownconventional systems and these benefits are not achievable by a humanbeing and require technology. The benefits include using the AI/ML/LLMtechnology to identify personal attacks in an audio piece of content inspite of the challenges discussed above that cannot be overcome by knownsystems and to generate a civility score.

FIG. 1 is a block diagram of a system 100 that incorporates a civilityscore engine 106B that identifies personal attacks and generates acivility score of each piece of audio content, such as podcasts, whereinthe generated civility score for each piece of audio content may bereturned to the user. The civility score may be displayed to a user aspart of search results, as part of a way to determine advertisementplacement in audio content or it may be displayed along with otherscores (like document quality scores and political leaning scores asdisclosed in U.S. patent application Ser. No. 18/220,437 filed Jul. 11,2023) in search results. Thus, the system 100 in FIG. 1 may be anadvertising system and company that uses the generated civility scoresfor podcasts in which the advertising system allows its customers toidentify podcasts for advertising wherein advertising money may bespent, for example, for podcasts with high civility scores indicatingfew or no personal attacks.

In each of the different possible alternatives of the system 100, a usermay use a computing device 102 to connect to, communicate with andaccess a backend system 106 over a communications path 104 in order toperform certain actions or services such as selecting audio content foradvertising campaigns or searches. Thus, the user connects to andcommunicates with the backend system 106, the system 106 performs itsfunctions (e.g., generates a civility score for each piece of audiocontent, a search that returns a civility score for each piece of audiocontent or generating an advertising campaign for a user that selectsaudio content with high civility scores, etc.) and returns results tothe user in a user interface generated by a user interface engine 106C.The civility score for each piece of audio content is generated in thesame manner for both the search function and the advertising campaignfunction as described below in more detail. Furthermore, although thecivility scoring engine 106B shown in FIG. 1 is part of the backendsystem 106, the civility scoring engine 106B may be an independentsystem that, for example, provides requested civility scores to thirdparties such as by using an API.

The system 100 may have a plurality of computing devices 102A, 102B,102C . . . , 102N that can each independently access the system 106 (toperform a search and see results with a civility score or set up and runadvertising campaigns and select audio pieces of content based on thecivility scores) over the communications path 104. Each computing devicemay have a processor, memory, wireless or wired connectivity circuits toconnect to the system 106 and a display wherein the memory stores aknown browser application, such as Google® Chrome®, etc., that is aplurality of lines of instructions executed by the processor that allowsthe user to interact with the system 106 in a known manner.Alternatively, the processor of each computing device 102 may execute amobile app or other application that is a plurality of lines ofinstructions executed by the processor that allows the user to interactwith the system 106 in a known manner. The system 106 may send back dataor HTML, pages with the search results (including civility scores) oradvertising campaign results (with civility scores) that are convertedinto a user interface by the browser or application and displayed on thedisplay of the computing device (examples of the user interface areshown in FIG. 5 .)

As shown in FIG. 1 , each computing device 102 may be a different devicesuch as a laptop computer 102A. a tablet computer 102B, a personalcomputer 102D, a smartphone device 102N or any other device that iscapable of connecting to and communicating with the backend system 106.The communications path 104 may be a wireless and/or wired path (or acombination of wired and wireless systems or networks) that may besecure or unsecure. Alternatively to the computer architecture shown inFIG. 1 , the system 106 or the civility scoring engine 106B may have oneor more Application Programming Interfaces (APIs) that provide thecivility scores of audio content as a software as a service to thirdparties.

The backend system 106 may be implemented by one or more computingresources, such as server computers, blade servers, cloud computingresources, etc. that have at least one processor and memory that storeand execute a plurality of lines of instructions/computer code toperform the search and scoring operations of the backend system 106. Thesystem 106, when the system is a search system, may further have asearch engine 106A, the civility scoring engine 106B and the userinterface engine 106C, each of which may be a plurality of lines ofinstructions/computer code executed by the processor of the computersystem that hosts the backend system 106. The search engine 106A mayperform the well known search engine operations to parse a keywordquery, perform the search and return the one or more pieces of contentthat form the search results in a well-known manner and with a civilityscore for each audio piece of content or return advertising campaignresults based on the civility scores of the audio content. The scoringengine 106B may generate, using a combination of ensemble AI/MLtechniques, a civility score for each piece of audio content, such as apodcast, that is discussed below in more detail with reference to FIG. 2. The user interface engine 106C collects the search results oradvertising campaign results and the civility score(s) for each piece ofaudio content and send those back to each computing device is responseto the request from each computing device in a well-known manner. Thebackend system 106 may have one or more hardware or software storagedevices 108A, . . . , 108N, that may be hardware or software or acombination of hardware and software, that store the data used for thesearches including the software for the various engines, user data, dataused to perform the civility assessments. The storage devices may alsostore the training data for the AI techniques, the resultant civilityscore for each piece of audio content, the audio content data, a set oflarge language model prompts and a search index that are used togenerate the civility score for each piece of audio content. In oneembodiment, the backend system 106 shown in FIG. 1 may be preferableimplemented using deep learning transformer models, large languagemodels and weak labeling algorithms, graphics processing unit (GPU)Hardware and search indices.

The civility scoring engine 106B (including various artificialintelligence (AI) technology) provide a technical solution to thetechnical problem of identifying personal attacks in audio pieces ofcontent in several novel ways and overcomes the limitations ofconventional systems and techniques as discussed above. The technologyleverages large language models, which possess a richer understanding ofmeaning, when compared to standard machine learning classificationsystems, to determine the meaning of a piece of audio content. Whendeciding whether a statement or discussion in a piece of audio content,such as a podcast, contains an attack, the civility scoring system 106Bconsiders the broader context, making it capable of betterdifferentiating innocuous statements from personal attacks. On the otherhand, the technology is also able to identify instances where seeminglyharmless words are used offensively. Importantly, the technology'sability to reason and navigate ambiguity, enables it to uncover indirectoffenses and to also discern when language is not offensive.

The technical solutions of the civility scoring are provided by anensemble of AI elements whose functions and operations cannot beperformed by a human being or in the human mind. The AI elements mayinclude proprietary processes for annotating, diversifying andde-noising large training sets which is crucial for accounting for thevariability in cultural interpretations and norms and LLM transformerarchitectures that focus not only on the parts of the text that maycontain attacks but also, the broader context. The AI elements mayfurther include proprietary processes for assigning a civility score toan entire episode and in turn, show and proprietary prompt engineeringtechniques that corroborates domain-expert knowledge and enable the AItechnology to align with the target task's objectives. Further detailsof the civility scoring and its AI aspects are now discussed withreference to FIGS. 2 and 3 .

FIG. 2 illustrates more details of the civility scoring engine 106B thatidentifies personal attacks in audio content, such as a podcast, andgenerates a civility score for each piece of audio content and FIG. 3illustrates a method 300 for generating a civility score for a piece ofaudio content. The civility scoring engine 106B may be part of andintegrated into a larger system 106 as shown in FIG. 1 or may be aservice/tool implemented on a computer system that performs the analysisof each piece of audio content to generate the civility scores which arethen available to third parties, such as via an API. The pieces of thecivility score engine 106B may each be implemented as a plurality oflines of computer code/instructions that are executed by a processor ofthe computer system that hosts the civility scoring engine 106B so thatthe processor is configured to perform the operations and functions ofthe civility scoring engine 106B. FIG. 2 shows the inference data flow(solid lines) and training data flow (dotted lines) that result in thegeneration of the civility score.

The civility scoring engine 106B may receive a corpus of audio content200 that may be, in one embodiment, a plurality of podcasts. Each of thepieces of audio content may be transcribed (process 302) to text using aknown and commercially available transcription service 201 and stored ina store 212 (that can be implemented as a hardware, software andhardware/storage database). Each piece of transcribed audio content mayconsist of a list of transcribed text chunks that are stored in thestore 212. In one implementation, during the transcription, each textualsegment corresponds to the specific part of the audio recording whereone person is speaking and thus isolates the parts of the audio where asingle person is contributing to the conversation.

The civility scoring engine 106B may perform a process 304 to generatetraining data that is then used to train the machine learning (ML) modelwhich is then used to identity any personal attacks and generate thecivility score for each piece of audio content. The process 304 mayannotate, diversify and de-noise large training sets which is crucialfor accounting for the variability in cultural interpretations andnorms. In one embodiment, the training data is generated using aplurality of large language models (LLMs) 204 (LLM1, LLM2, LLM 3, . . .LLMN as shown in FIG. 2 ). Thus, during the training data generationprocess, the collection of LLMs 204 are invoked using a collection ofdifferent LLM prompts 202 that may be expert crafted prompts to elicitinformation from the LLMs. Each LLM is instructed to label text chunksgiven a broader context since the labeling and subsequent scoringprocesses take into account a broader window of text around a specifictext chunk being labeled or scored so that seemingly innocuousstatements are not classified as attacks and also, subtle personalattacks are detected and classified as such. The collection of LLMprompts may be a plurality of prompts generated using known promptengineering techniques such as chain-of-thought prompting,tree-of-thoughts prompting and others. An example prompt that may beused in the process is shown in FIG. 4 . The novel prompt engineeringtechniques may corroborate domain-expert knowledge and enable the AItechnology to align with the target task's objectives. In oneembodiment, domain expert knowledge is injected into the LLMs viaadvanced prompting techniques and fine-tuning (examples of the promptsare shown in FIG. 4 ).

Once each LLM labels each text chunk for an audio piece of content, astore 205 (hardware or software database) stores the weakly labeled datathat results from each LLM and each prompt. The weakly labeled data maybe input to an aggregation process 206 whose output is a final label foreach text chunk in the piece of audio content, such as a podcast. Theaggregation process 206 combines all of the labeled text chunks from allof the LLMS into a final label. As shown in FIG. 2 , the final label foreach text chunk for each piece of audio content may be subject to humanvalidation 218 and then stored in a training data store 208. Using thefinal labels stored in the training data database 208, the transcribedaudio content 212 and the transcription service 201 data, a training job214 is used to train (process 306 in FIG. 3 ) the civility machinelearning model 216. In one embodiment, the civility model 216 may be aknown transformer model that is trained using the labeled text chunkdata generated using the LLMs 204 and aggregation process 206, thetranscribed audio content 212 and the transcription service 201 data.Note that the training of the ML model may be performed once and thenthe ML model may be used to generate civility scores, although thetraining of the ML model may be updated periodically.

Once the civility model 216 is trained, the civility model (such as atransformer model) is used generate a civility score (220) for eachpiece of audio content to score each piece of audio content, such as apodcast, and assigns a civility score (308) for each transcribed pieceof audio content. During the civility score determining, the process 308may assign a civility score to an entire episode and in turn, an entireshow that may then be shown to the user as shown in FIG. 5. The civilityscore for each piece of audio content indicates a quantity of personalattacks contained in each piece of audio content with a high civilitypiece of audio content having few or no personal attacks and thus beingvery civil while a low civility piece of audio content has a pluralityof personal attacks.

In more detail, during the scoring process 220, each text chunk can becategorized as 1) not containing any person attack (non-attack); 2)containing a personal attack (attack); or 3) containing a strongpersonal attack (severe attack) based on the classification given by thecivility model. Subsequently, a range of algorithms can be deployed toobtain the civility score for each episode. These algorithms works basedon number of attacks and severe attacks normalized by the duration ofeach episode. It is understood by those skilled in the art that variousdifferent algorithms may be used to perform the scoring. In oneembodiment, statistical methods are used to assess a frequency ofpersonal attacks in relation to the entire range of podcasts.

An indexing process/job 222 writes the scored audio content (textchunks, episodes and/or shows) and scores into a search index/databaseso that the civility scores may be retrieved during a search andprovided to a user. While FIG. 2 illustrates generating and storing thecivility scores prior to a search or civility score request, the system106B may also perform the civility score generation on the fly asneeded.

FIG. 5 illustrates an example of a user interface 500 generated by thesystem that displays various podcasts, a civility score and a scorechange. In the example, the plurality of pieces of audio content arepodcasts and the user interface shows a top 10 list of podcasts to auser along with the civility scores. The user interface may include acivility score portion 502 and a score change portion 504 for eachpodcast. Each podcast may be assigned a “low civility”, “mediumcivility” or a “high civility” rating in the score portion 502. A usercan review the civility score and select a podcast (during a search) orselect a podcast for an advertising campaign. In addition to thecivility score, the user interface may display a score change since lastepisode in the score change portion 504. The score change may be “nochange”, “negative change” or “positive change” and each may indicatewhen the podcast or audio content was aired.

The foregoing description, for purpose of explanation, has been withreference to specific embodiments. However, the illustrative discussionsabove are not intended to be exhaustive or to limit the disclosure tothe precise forms disclosed. Many modifications and variations arepossible in view of the above teachings. The embodiments were chosen anddescribed in order to best explain the principles of the disclosure andits practical applications, to thereby enable others skilled in the artto best utilize the disclosure and various embodiments with variousmodifications as are suited to the particular use contemplated.

The system and method disclosed herein may be implemented via one ormore components, systems, servers, appliances, other subcomponents, ordistributed between such elements. When implemented as a system, suchsystems may include and/or involve, inter alia, components such assoftware modules, general-purpose CPU, RAM, etc. found ingeneral-purpose computers. In implementations where the innovationsreside on a server, such a server may include or involve components suchas CPU, RAM, etc., such as those found in general-purpose computers.

Additionally, the system and method herein may be achieved viaimplementations with disparate or entirely different software, hardwareand/or firmware components, beyond that set forth above. With regard tosuch other components (e.g., software, processing components, etc.)and/or computer-readable media associated with or embodying the presentinventions, for example, aspects of the innovations herein may beimplemented consistent with numerous general purpose or special purposecomputing systems or configurations. Various exemplary computingsystems, environments, and/or configurations that may be suitable foruse with the innovations herein may include, but are not limited to:software or other components within or embodied on personal computers,servers or server computing devices such as routing/connectivitycomponents, hand-held or laptop devices, multiprocessor systems,microprocessor-based systems, set top boxes, consumer electronicdevices, network PCs, other existing computer platforms, distributedcomputing environments that include one or more of the above systems ordevices, etc.

In some instances, aspects of the system and method may be achieved viaor performed by logic and/or logic instructions including programmodules, executed in association with such components or circuitry, forexample. In general, program modules may include routines, programs,objects, components, data structures, etc. that perform particular tasksor implement particular instructions herein. The inventions may also bepracticed in the context of distributed software, computer, or circuitsettings where circuitry is connected via communication buses, circuitryor links. In distributed settings, control/instructions may occur fromboth local and remote computer storage media including memory storagedevices.

The software, circuitry and components herein may also include and/orutilize one or more type of computer readable media. Computer readablemedia can be any available media that is resident on, associable with,or can be accessed by such circuits and/or computing components. By wayof example, and not limitation, computer readable media may comprisecomputer storage media and communication media. Computer storage mediaincludes volatile and nonvolatile, removable and non-removable mediaimplemented in any method or technology for storage of information suchas computer readable instructions, data structures, program modules orother data. Computer storage media includes, but is not limited to, RAM,ROM, EEPROM, flash memory or other memory technology, CD-ROM, digitalversatile disks (DVD) or other optical storage, magnetic tape, magneticdisk storage or other magnetic storage devices, or any other mediumwhich can be used to store the desired information and can accessed bycomputing component. Communication media may comprise computer readableinstructions, data structures, program modules and/or other components.Further, communication media may include wired media such as a wirednetwork or direct-wired connection, however no media of any such typeherein includes transitory media. Combinations of the any of the aboveare also included within the scope of computer readable media.

In the present description, the terms component, module, device, etc.may refer to any type of logical or functional software elements,circuits, blocks and/or processes that may be implemented in a varietyof ways. For example, the functions of various circuits and/or blockscan be combined with one another into any other number of modules. Eachmodule may even be implemented as a software program stored on atangible memory (e.g., random access memory, read only memory, CD-ROMmemory, hard disk drive, etc.) to be read by a central processing unitto implement the functions of the innovations herein. Or, the modulescan comprise programming instructions transmitted to a general-purposecomputer or to processing/graphics hardware via a transmission carrierwave. Also, the modules can be implemented as hardware logic circuitryimplementing the functions encompassed by the innovations herein.Finally, the modules can be implemented using special purposeinstructions (SIMD instructions), field programmable logic arrays or anymix thereof which provides the desired level performance and cost.

As disclosed herein, features consistent with the disclosure may beimplemented via computer-hardware, software, and/or firmware. Forexample, the systems and methods disclosed herein may be embodied invarious forms including, for example, a data processor, such as acomputer that also includes a database, digital electronic circuitry,firmware, software, or in combinations of them. Further, while some ofthe disclosed implementations describe specific hardware components,systems and methods consistent with the innovations herein may beimplemented with any combination of hardware, software and/or firmware.Moreover, the above-noted features and other aspects and principles ofthe innovations herein may be implemented in various environments. Suchenvironments and related applications may be specially constructed forperforming the various routines, processes and/or operations accordingto the invention or they may include a general-purpose computer orcomputing platform selectively activated or reconfigured by code toprovide the necessary functionality. The processes disclosed herein arenot inherently related to any particular computer, network,architecture, environment, or other apparatus, and may be implemented bya suitable combination of hardware, software, and/or firmware. Forexample, various general-purpose machines may be used with programswritten in accordance with teachings of the invention, or it may be moreconvenient to construct a specialized apparatus or system to perform therequired methods and techniques.

Aspects of the method and system described herein, such as the logic,may also be implemented as functionality programmed into any of avariety of circuitry, including programmable logic devices (“PLDs”),such as field programmable gate arrays (“FPGAs”), programmable arraylogic (“PAL”) devices, electrically programmable logic and memorydevices and standard cell-based devices, as well as application specificintegrated circuits. Some other possibilities for implementing aspectsinclude: memory devices, microcontrollers with memory (such as EEPROM),embedded microprocessors, firmware, software, etc. Furthermore, aspectsmay be embodied in microprocessors having software-based circuitemulation, discrete logic (sequential and combinatorial), customdevices, fuzzy (neural) logic, quantum devices, and hybrids of any ofthe above device types. The underlying device technologies may beprovided in a variety of component types, e.g., metal-oxidesemiconductor field-effect transistor (“MOSFET”) technologies likecomplementary metal-oxide semiconductor (“CMOS”), bipolar technologieslike emitter-coupled logic (“ECL”), polymer technologies (e.g.,silicon-conjugated polymer and metal-conjugated polymer-metalstructures), mixed analog and digital, and so on.

It should also be noted that the various logic and/or functionsdisclosed herein may be enabled using any number of combinations ofhardware, firmware, and/or as data and/or instructions embodied invarious machine-readable or computer-readable media, in terms of theirbehavioral, register transfer, logic component, and/or othercharacteristics. Computer-readable media in which such formatted dataand/or instructions may be embodied include, but are not limited to,non-volatile storage media in various forms (e.g., optical, magnetic orsemiconductor storage media) though again does not include transitorymedia. Unless the context clearly requires otherwise, throughout thedescription, the words “comprise,” “comprising,” and the like are to beconstrued in an inclusive sense as opposed to an exclusive or exhaustivesense; that is to say, in a sense of “including, but not limited to.”Words using the singular or plural number also include the plural orsingular number respectively. Additionally, the words “herein,”“hereunder,” “above,” “below,” and words of similar import refer to thisapplication as a whole and not to any particular portions of thisapplication. When the word “or” is used in reference to a list of two ormore items, that word covers all of the following interpretations of theword: any of the items in the list, all of the items in the list and anycombination of the items in the list.

Although certain presently preferred implementations of the inventionhave been specifically described herein, it will be apparent to thoseskilled in the art to which the invention pertains that variations andmodifications of the various implementations shown and described hereinmay be made without departing from the spirit and scope of theinvention. Accordingly, it is intended that the invention be limitedonly to the extent required by the applicable rules of law.

While the foregoing has been with reference to a particular embodimentof the disclosure, it will be appreciated by those skilled in the artthat changes in this embodiment may be made without departing from theprinciples and spirit of the disclosure, the scope of which is definedby the appended claims.

What is claimed is:
 1. A system, comprising: a computer system having aprocessor and a plurality of lines of instructions that are executed bythe processor that is configured to: retrieve a plurality of pieces ofaudio content; generate, for each piece of audio content using a trainedmachine learning model, a civility score wherein the civility score foreach piece of audio content indicates a quantity of personal attacks ineach piece of audio content; and display, on a display of a usercomputing device connected to the computer system, a civility score forone or more pieces of audio content.
 2. The system of claim 1, whereinthe trained machine learning model is a transformer model.
 3. The systemof claim 2, wherein the processor is further configured to generate,using the plurality of pieces of audio content, a set of training datato train the transformer model.
 4. The system of claim 3, wherein theprocessor is further configured to invoke a plurality of large languagemodels to generate a label for each portion of each piece of audiocontent and perform weak labeling of the generated labels to generate alabel for each piece of audio content.
 5. The system of claim 1, whereinthe processor is further configured to transcribe each piece of audiocontent, generate a plurality of text chunks for each piece of audiocontent and generate a civility score for each text chunk for each pieceof audio content.
 6. The system of claim 1, wherein each piece of audiocontent is a podcast.
 7. A method, comprising: retrieving, by a computersystem having a processor, a plurality of pieces of audio content;generating, by a trained machine learning model executed by theprocessor of the computer system, for each piece of audio content, acivility score wherein the civility score for each piece of audiocontent indicates a quantity of personal attacks in each piece of audiocontent; and displaying, on a display of a user computing deviceconnected to the computer system, a civility score for one or morepieces of audio content.
 8. The method of claim 7, wherein the trainedmachine learning model is a transformer model.
 9. The method of claim 8further comprising generating, by the computer system using theplurality of pieces of audio content, a set of training data to trainthe transformer model.
 10. The method of claim 9 further comprisingtraining the trained transformer model that further comprises invoking aplurality of large language models to generate a label for each portionof each piece of audio content and performing weak labeling of thegenerated labels to generate a label for each piece of audio content.11. The method of claim 7 further comprising transcribing each piece ofaudio content, generating a plurality of text chunks for each piece ofaudio content and wherein generating the civility score furthercomprises generating a civility score for each text chunk for each pieceof audio content.
 12. The method of claim 7, wherein each piece of audiocontent is a podcast.